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Big Can Be Great: 

Enhancing Undergraduate Education 
at Research-Extensive Universities 

Although recent results from the National Survey of Student Engagement (NSSE) 
suggest there is considerable variation in levels of undergraduate student engagement 
within Carnegie institutional classifications, research also suggests different 
classifications of institutions exhibit different types (Pike and Kuh, 2005) and levels 
(Pike, Kuh and Gonyea, 2003) of student engagement. However, an important question 
that requires additional examination is the potential relationship between particular 
engagement variables and various outcomes of interest to students and institutions. 
Given the time and resource limitations that faculty, staff, and administrators face in their 
efforts to improve academic quality, analyses that move beyond norm-based or statistical 
comparisons and test empirical relationships within theory-based models of quality in 
undergraduate education may provide a more effective means to focus improvement 
efforts. This study examines student-reported learning and development factor scores, 
the likelihood of persistence, and students’ ratings of their overall educational experience 
based on NSSE responses from over 1,500 first-year and senior students in 2004 at a 
large, research-extensive university. Using Chickering and Gamson’s (1987) “Seven 
Principles for Effective Practice in Undergraduate Education” as a conceptual framework 
to guide variable selection, the results of four models suggest that a high level of 
academic challenge, institutional emphasis on academic work and studying, and prompt 
feedback on academic performance may provide the most productive avenues for 
enhancing the impact and quality of undergraduate education at large, research-extensive 
universities. Possible explanations and potential implications of these and other findings 
for students, faculty members, institutional leaders, policymakers, and accrediting bodies 
are discussed. 



KEYWORDS: Student engagement, student persistence, student development, student 
learning, undergraduate education, college impact, academic quality, best practices 
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INTRODUCTION 

Student engagement is a well-established concept in the research literature on 
higher education quality, student development, and student learning (Pike, 2004; Kuh, 
2002; Kuh, 2001; Ku and Hu, 2001a; NSSE, 2001). Its impact is most evident in the 
creation and growth of the National Survey of Student Engagement (NSSE) as an 
evaluative instrument designed to gauge best practices and experiences that higher 
education research indicates are important to enhancing student learning and 
development (Astin, 1993; Kuh, 2003; Pace, 1984; Pascarella and Terenzini, 1991). For 
those are dissatisfied with the utility and insight gained from traditional “resource and 
reputation” models of higher education quality and concerned with trends in the 
accountability movement, paying greater attention to best practices that enhance student 
learning represents a welcome development in the quest to better define and evaluate 
academic quality. In fact, a series of recent issues of the well-known college ranking 
guide published by U.S. News and World Report (USNWR) have published voluntary 
reports of specific NSSE results from some colleges and universities (USNWR’s 
“America’s Best Colleges”, 2002-2005) . These results included student reports of the 
extent to which they participated in or experienced various activities that can enhance 
student learning and development such as asking questions in class, receiving prompt 
feedback on coursework, having discussions with instructors outside of class, writing 
papers, and participating in research. 

The theoretical roots of the student engagement as an important factor in student 
learning and development are strong as well. For many years, researchers and higher 
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education leaders have given a great deal of attention to the concepts of student 
involvement (Astin, 1993), faculty-student interaction (Pascarella and Terenzini, 1991), 
and student academic and social integration (Spady, 1971; Tinto, 1975; Tinto, 1993) as 
explanations for the impact of college on students and as predictors of student persistence 
and development. Although the concept of student engagement contains some 
complementary and common characteristics with these concepts, student engagement and 
NSSE are anchored most directly in Chickering and Gamson’s (1987) “Seven Principles 
for Effective Practice in Undergraduate Education” and focus on a fairly defined set of 
educational interactions and experiences among students, faculty members, and their 
institutions. These principles are: 

1. Student-faculty contact 

2. Cooperation among students 

3. Active learning 

4. Prompt feedback 

5. Time on task 

6. Communication of high expectations 

7. Respect for diverse talents and ways of learning 

However, in spite of the increased attention being given to student engagement 
as a determinant and indicator of undergraduate education quality, leaders and 
researchers have focused less attention to the examination of the particular issues that 
different kinds of institutions face in their efforts to enhance undergraduate education, 
including large, research-extensive universities. Although recent results from the 
National Survey of Student Engagement (NSSE) suggest there is considerable variation 
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in levels of undergraduate student engagement within Carnegie institutional 
classifications, research also suggest different classifications of institutions exhibit 
different types (Pike and Kuh, 2005) and levels (Pike, Kuh and Gonyea, 2003; Umbach 
and Wawrzynski, 2005) of student engagement (see Figure 1). However, an important 
question that requires additional examination is the potential relationship between 
particular engagement variables and various outcomes of interest to students, institutions, 
and external constituencies. Given the time and resource limitations that faculty, staff, 
and administrators face in their efforts to improve academic quality, analyses that move 
beyond norm-based or statistical comparisons and test empirical relationships based on 
theory-based models of quality in undergraduate education may provide a more effective 
means to focus improvement efforts. Furthermore, the literature has paid even less 
attention to how specific practices and experiences that shape student engagement may be 
related to and support a variety of desired institutional and student goals and whether or 
not certain practices may be more critical to such outcomes. The issue of whether of not 
there may be “synergies of engagement and best practice” across a variety of institutional 
and student goals also begs to be examined. 

PURPOSES AND RESEARCH QUESTIONS 

This study seeks to extend the range of study of student engagement, college 
impacts on students, and academic quality and to address some gaps in the research 
literature by examining student engagement variables and their relationship to 
institutional impacts on student learning, student development, student persistence, and 
student ratings of the overall educational experience. 
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The results of this study may be useful to both researchers of and leaders at large, 
research-extensive universities as both groups seek to understand and enhance academic 
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FIGURE 1. Student-Faculty Interaction Senior Benchmark Scores 
by Carnegie Classification 

Benchmark Scores Seniors 




Source: Student Pathways to Collegiate Success, NSSE 2004 Annual Survey Results 
(http://www.iub.edu/~nsse/2004 annual report/pdf/annual report.pdf) . 
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quality within the context of a complex institutional environment defined by multi- 
faceted missions and aims in teaching, research, and service. In addition, this study 
examines the utility of exploring student engagement and the impact of the college 
experience within a specific institution type. Differences in institutional characteristics 
such as size and mission may suggest unique issues and needs related to enhancing the 
impact of academic programs (Toutkoushian and Smart, 2001). Therefore, faculty, 
administrators, and students at large, research-extensive institutions may need to focus on 
specific strategies to enhance institutional impact, student outcomes, and academic 
quality. 



Finally, there are competing views regarding how to define and improve academic 
quality. One approach can be described as a business-centered “customer service” view 
of institutional quality that seeks to enhance students’ satisfaction with their educational 
experience. Efforts to “raise the bar” via high expectations and rigorous academic 
requirements characterize a second general approach. Simply put, do efforts to increase 
student satisfaction and academic challenge represent incompatible, competing, and even 
contradictory aims? This study seeks to examine this question in more detail and attempt 
to assess the extent to which these aims actually work against each other. 

Given these overall purposes, this study sought to answer the following questions: 

1) What is the relationship between various components of student engagement 
and a) student learning b) student development c) student retention and d) 
student ratings of their entire educational experience? 

2) What particular student engagement practices are most important? 

3) Do attempts to enhance student satisfaction and increase academic 
expectations of students represent competing and contradictory aims? 

4) What are the theory, practice, and policy implications for researchers, leaders, 
faculty members, and undergraduate students at large, research-extensive 
universities? 
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DATA AND METHODOLOGY 

This study employed a non-experimental research design and utilized both 
ordinary least-squares (OLS) regression and ordinal regression to examine and test NSSE 
survey results collected from over 1,500 first-year and senior students at a large, 
research-extensive university during Spring 2004. Overall, there were 2,012 respondents 
representing a response rate of 34 percent with a sampling error of ±2 percent. The 
overall respondents also reflect larger percentages of minority students compared to the 
overall population of first-year and senior students due to intentional over-sampling of 
minority students. The sample and respondent characteristics are provided in Table 1. 

The NSSE survey questionnaire itself has been carefully designed and 
constructed. The instrument’s conceptual framework and psychometric properties exhibit 
strong validity and reliability based on both statistical tests in addition to input and feed 
back from various experts and students (Kuh, 2003; Pike and Kuh, 2005). These results 
meet or exceed established standards for reliability and validity. In addition, the 
questionnaire is designed to minimize factors that are to known to adversely affect the 
validity of responses, especially in light of potential challenges associated with self- 
reports by students (Pike 1993, 1995, 1996, 1999). 

In order to explore multiple and distinct dimensions of institutional impact and 
academic quality, four models - two employing OLS regression and two employing 
ordinal regression - were developed. For all four models, the same independent variables 
were included based on Chickering and Gamson’s (1987, 1991) “Seven Principles for 
Good Practice in Undergraduate Education” as a conceptual and theoretical guide to 
enhance model parsimony and completeness. These principles were developed based on 
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a review of over 50 years of research and maintain strong influence in the practice and 
study of undergraduate education. 

Next, appropriate indicators of the various dimensions of institutional impact and 
academic quality were identified. In order to gauge institutional contribution to student 
learning and development, principle components analysis with varimax rotation was used 
to identify the structure of the institutional contribution/outcome items in the NSSE 
survey. The results indicated two distinct factors: one centered on academic skills and 
one centered on personal values and attitudes (the items that comprise each factor are 
listed in Table 3). NSSE items with a factor loading of .6 or higher were included in each 
factor and factor scores were calculated for each respondent. These two factors, labeled 
LRNGAIN and DEVELOP, were used respectively as the dependent variables for the 
two OLS models. The third and fourth models sought to examine relationships between 
the same set of independent variables and proxy indicators of student satisfaction: the 
likelihood a student would return to the same institution if starting over (SAMECOLL) 
and a rating of the overall quality of the educational experience (ENITEEXP). Due to the 
ordinal nature of the responses to these items, ordinal regression was employed for these 
models. All four models were estimated using SPSS 12.0 for Windows. 

Recognizing the impact that particular and generally immutable characteristics of 
students and their backgrounds can have on their college experience and the impact of 
that experience (Astin, 1993; Bean, 1980; Braxton, 2000; Pascarella and Terenzini, 1991; 
Tinto, 1975 and 1993) each model controlled for academic preparation/background using 
ACT and SAT-converted test scores (Marco, G.L., Abdel-fattah, A.A., and Baron, P.A., 
1992), gender, minority status, full-time/part-time enrollment status, class rank, place of 
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residence, whether or not the student started college at the current institution or 
transferred from another post- secondary institution, and self-reports of the majority of 
grades earned in classes. All of the variables used in each of the four models, including 
their respective question item prompts and coding, are listed in Table 2. 
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Table 1. Sample Characteristics and Response Rate 

(percentages within each category) 





FY 


ALL SR 


Response Rate 3 


Overall 




34% 


By Class 


36% 


32% 


NSSE Sample Size b 


2,426 


3,537 


Sample Error 3 


Overall 




2.0% 


By Class 


3.0% 


2.7% 


Number of Respondents 13 


882 


1,139 


Total Population 


6.003 


10,597 


Student Characteristics 31 
Mode of Completion 


Paper 


0% 


0% 


Web 


100% 


100% 


Gender 


Female 


60% 


58% 


Male 


40% 


42% 


Race/Ethnicity 


African American/Black 


15% 


13% 


Am. Indian/Native American 


1% 


1% 


Asian Am./Pacific Islander 


13% 


27% 


Cauc asian/White 


60% 


48% 


Hispanic or Latino 


3% 


4% 


Other 


0% 


1% 


Multi-racial 


8% 


7% 


International/Foreign National 


7% 


17% 


Class Level 


44% 


56% 


Enrollment Status 


Full Time 


99% 


90% 


Part Time 


1% 


10% 


Place of Residen ce 


On-campus 


86% 


8% 


Off-campus 


14% 


92% 


Transfer Status 


Transfer Students 


6% 


33% 


Age 


Non-Traditional (24 or older) 


0% 


23% 


Traditional (less than 24) 


100% 


77% 



Source: NSSE Institutional Respondent Characteristics Report, 2004. 
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TABLE 2. Independent, Dependent, and Control Variables 

In your experience at your institution during the current school year, about how often have you done 
each of the following? l=never, 2=sometimes, 3=often, 4=very often 

INDEPENDENT VARIABLES VARIABLE NAMES 

1. Asked questions in class or contributed to class discussions CLQUEST 

2. Made a class presentation CLSPRESEN 

3. Prepared two or more drafts of a paper or assignment REWROPAP 

before turning it in 

4. Worked on a paper or project that required integrating ideas or INTEGRAT 

information from various sources 

5. Included diverse perspectives (different races, religions, genders, DIVCLASS 

political beliefs, etc.) in class discussions or writing assignments 

6. Worked with other students on projects during class CLASSGRP 

7. Worked with classmates outside of class to prepare class OCCGRP 

assignments 

8. Put together ideas or concepts from different courses when INTIDEAS 

completing class assignments or during class discussions 

9. Tutored or taught other students (paid or voluntary) TUTOR 

10. Participated in a community-based project as part of a regular COMMPROJ 

course 

11. Used an electronic medium (list-serve, chat group, Internet, etc.) ITACADEM 

to discuss or complete an assignment 

12. Used e-mail to communicate with an instructor EMAIL 

13. Discussed grades or assignments with an instructor FACGRADE 

14. Talked about career plans with a faculty member or advisor FACPLANS 

15. Discussed ideas from your readings or classes with faculty members FACIDEAS 
outside of class 

16. Received prompt feedback from faculty on your academic FACFEED 

performance (written or oral) 

17. Worked harder than you thought you could to meet an instructor's WORKHARD 
standards or expectations 

18. Worked with faculty members on activities other than coursework FACOTHER 
(committees, orientation, student life activities, etc.) 

19. Discussed ideas from your readings or classes with others outside OOCIDEAS 
of class (students, family members, coworkers, etc.) 

20. Had serious conversations with students of a different race or DIVSTUD 

ethnicity than your own 

21. Had serious conversations with students who are very different DIFFSTU2 
from you in terms of their religious beliefs, political opinions, or 

personal values 

During the current school year, how much has your coursework emphasized the following mental 
activities? l=very little, 2=some, 3=quite a bit, 4=very much 

22. Analyzing the basic elements of an idea, experience, or theory, ANALYZE 

such as examining a particular case or situation in depth and 

considering its components 

23. Synthesizing and organizing ideas, information, or experiences into SYNTHESZ 
new, more complex interpretations and relationships 

24. Making judgments about the value of information, arguments, or EVALUATE 
methods, such as examining how others gathered and interpreted data 

and assessing the soundness of their conclusions 

25. Applying theories or concepts to practical problems or in new APPLYING 
Situations 
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(Table 2 continued) 
l=very little to 7=very much 

INDEPENDENT VARIABLES VARIABLE NAMES 

26. To what extent have your examinations during the current school EXAMS 

year challenged you to do your best work? 

During the current school year, about how much reading and writing have you done? 
l=none, 2=between 1 and 4, 3=between 5 and 10, 4=between 11 and 20, 5=more than 20 

27. Number of assigned textbooks, books, or book-length packs of READASGN 

course readings 

28. Number of written papers or reports of 20 pages or more WRITEMOR 

29. Number of written papers or reports between 5 and 19 pages WRITEMID 

30. Number of written papers or reports of fewer than 5 pages WRITESML 

In a typical week, how many homework problem sets do you complete? 
l=none, 2=1-2, 3=3-4, 4=5-6, 5=more than 6 

31. Number of problem sets that take you more than an hour to complete PROBSETA 

32. Number of problem sets that take you less than an hour to complete PROBSETB 

Which of the following have you done or do you plan to do before you graduate from your 

institution? 

Recoded: 1= do not plan to do, 2=have not decided, 3=plan to do, 4=done 

33. Practicum, internship, field experience, co-op experience, or clinical INTERN 
assignment 

34. Participate in a learning community or some other formal program LEANRCOM 
where groups of students take two or more classes together 

35. Work on a research project with a faculty member outside of course RESEARCH 
or program requirements 

36. Study abroad STUDYABR 

37. Independent study or self-designed major INDSTUDY 

3 8. Culminating senior experience (comprehensive exam, capstone SENIORX 
course, thesis, project, etc.) 

About how many hours do you spend in a typical 7-day week doing each of the following? 

1=0 hrs/wk, 2=1-5 hrs/wk, 3=6-10 hrs/wk, 4=11-15 hrs/wk, 5=16-20 hrs/wk, 6=21-25 hrs/wk, 7=26-30 

hrs/wk, 8=more than 30 hrs/wk 

39. Preparing for class ACADPR01 

40. Participating in co-curricular activities COCURR01 



To what extent does your institution emphasize each of the following? 
l=very little, 2=some, 3=quite a bit, 4=very much 

41. Spending significant amounts of time studying and on academic ENVSCHOL 
work 
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(Table 2 continued) 

CONTROL VARIABLES 

42. Did you begin college at your current institution or elsewhere? 
(dummy-coded) 

43Thinking about this current academic term, how would you 

characterize your enrollment? (dummy-coded: full-time/part-time) 

44. What have most of your grades been up to now at this institution? 

( 1 = C- or lower. . .8 = A) 

45. Which of the following best describes where you are living now 
while attending college? 

(Recoded: 1 = On campus 2 = Off campus within walking 3 = Off 
campus within driving distance) 

46. Gender (dummy-coded) 

47. Class Rank (dummy-coded: first year/senior) 

48. Academic preparation (ACT or SAT-converted score) 

49. Minority status (dummy-coded white/non-white) 

DEPENDENT VARIABLES/FACTORS 

50. Learning factor score* 

51. Development factor score** 

52. If you could start over again, would you go to the same institution 
you are now attending? (1= Definitely no, 2 = Probably no, 3 = 
Probably yes, 4 = Definitely yes) 

53. How would you evaluate your entire educational experience 
at this institution? (1 = Poor, 2 = Fair, 3 = Good, 4 = Excellent) 



VARIABLES NAMES 
ENTER 

ENRLMENT 

GRADES 04 



LIVENOW 

GENDER 

CLASSRAN 

ACTSAT 

MINORITY 

LRNGAIN 

DEVELOP 

SAMECOLL 



ENTIREEXP 



*LRNGAIN factor items: acquired broad general education, writing skills, speaking skills, critical thinking 
skills, quantitative analysis skills, IT skills, team skills 

**DEVELOP factor items: learning on my own, understanding myself, understanding other 
races/ethnicities, personal values/ethics, contributing to community, deeper sense of my spirituality 
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RESULTS 

The following tables present the statistical results from the four models. Table 3 
provides summary statistics including adjusted/pseudo R 2 , F/% 2 statistics, model 
significance, and sample sizes (n). Tables 4, 5, 6, and 7 provide the regression output for 
LRNGAIN, DEVELOP, SAMECOLL, and ENTIREXP respectively. These results 
include parameter estimates/variable coefficients and confidence intervals for the control 
variables and independent variables of interest. Regression diagnostics for the OLS 
models did not suggest problems with collinearity among the independent variables or the 
distribution of residuals (Gujarati, 1995). 
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Table 3. Model Summaries 



Model 


Adiusted/Pseudo R 2 * 


J y2 ^ 


Significance 


n 


LRNGAIN 


.389 


22.642 


.000 


1,528 


DEVELOP 


.228 


11.061 


.000 


1,529 


SAMECOLL 


.145* 


240.756** 


.000 


1,532 


ENTIREEXP 


.232* 


405.257** 


.000 


1,533 
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Table 4. LRNGAIN Model Results 

Coefficients 



Model 


Unstandardized 

Coefficients 


Standardized 

Coefficients 


t 


Sig. 


Collinearitv Statistics 


B 


Std. Error 


Beta 


Tolerance 


VI F 


1 (Constant) 


-3.961 






-12.121 








enter 


-.054 




-.015 


-.718 


.473 


.939 


1.065 


enrlment 


-.005 


.103 


-.001 


-.046 


.963 


.877 


1.141 


grades04* 


.028 


.014 


.048 


2.051 


.040 


.739 


1.354 


livenow 


-.021 


.035 


-.018 


-.611 


.541 


.473 


2.116 


gender 


-.062 




-.031 


-1.386 


.166 


.818 


1.222 


classran** 


.063 




.096 


3.056 




.406 


2.463 


minority 


.003 




.001 


.062 


.951 


.865 


1.156 


actsat 


.000 




-.030 


-1.212 


.226 


.664 


1.507 


email** 


.088 




.073 


2.775 


.006 


.578 


1.729 


facgrade 


-.012 




-.011 


-.392 


.695 


.548 


1.824 


facplans 


.016 




.015 


.589 


.556 


.642 


1.559 


facideas 


-.004 




-.003 


-.136 


.891 


.612 


1.633 


facother 


.039 




.034 


1.354 


.176 


.639 


1.565 


research 


-.027 




-.028 


-1.167 


.243 


.712 


1.405 


divclass 


.038 




.036 


1.560 


.119 


.744 


1.344 


classgrp 


.035 




.031 


1.404 


.161 


.820 


1.220 


occgrp*** 


.098 




.092 


3.898 


.000 


.714 


1.400 


tutor 


-.002 




-.002 


-.079 


.937 


.750 


1.334 


oocideas* 


.061 


.026 


.056 


2.339 


.019 


.689 


1.451 


divrstud 


-.015 


.030 


-.015 


-.492 




.433 


2.307 


diffstu2 


-.027 


.030 


-.027 


-.887 




.442 


2.261 


learncom 


.022 


.019 


.025 


1.152 




.841 


1.189 


clquest 


.020 




.017 


.694 


.488 


.652 


1.534 


clpresen 


.016 


.027 


.014 


.583 


.560 


.659 


1.517 


commproj 


-.037 


.027 


-.032 


-1.370 


.171 


.756 


1.324 


itacadem* 


.049 


.021 


.052 




.021 


.795 


1.258 


intern 


.010 


.026 


.009 


.405 


.686 


.853 


1.172 


studyabr 


.020 


.021 




.938 


.348 


.857 


1.167 


indstudy 


-.003 


.023 


-.003 


-.143 


.887 


.836 


1.196 


seniorx* 


.047 


.023 


.045 


2.038 


.042 


.816 


1.225 


facfeed*** 


.107 


.027 


.093 


3.887 


.000 


.706 


1.417 


envschol*** 


.288 


.027 


.235 


10.580 


.000 


.808 


1.238 


workhard 


.005 




.005 


.191 


.849 


.642 


1.558 


memorize 


-.021 




-.018 


-.871 


.384 


.900 


1.111 


analyze 


.047 


hi 


.036 


1.379 


.168 


.580 


1.725 


synthesz*** 


.105 


.031 


.092 






.528 


1.894 


evaluate** 


.080 


.029 


.074 


2.782 


.005 


.565 


1.771 


applying*** 


.142 


.030 


.123 


4.719 


.000 


.587 


1.705 


exams*** 


.105 


.019 


.125 


5.437 


.000 


.751 


1.331 


readasgn 


.019 


.022 


.019 


.866 


.387 


.833 


1.200 


writemor 


-.012 


.037 


-.007 


-.315 


.753 


.785 


1.275 


writemid 


.015 


.026 


.014 


.574 


.566 


.701 


1.426 


writesml 


.001 


.021 


.001 


.050 


.960 


.777 


1.286 


probseta 


-.029 


.020 


-.033 


-1.431 


.153 


.767 


1.304 


probsetb 


.032 


.019 


.038 


1.689 


.091 


.808 


1.238 
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Model 

1 (Constant) 

enter 
enrlment 
grades04** 
livenow*** 
gender 
classran 
minority 
actsat** 
email* 
facgrade** 
facplans 
facideas 
facother* 
research 
divclass*** 
classgrp 
occgrp 
tutor 
oocideas 
divrstud 
diffstu2*** 
learncom 
olquest 
clpresen 
commproj** 
itacadem 
intern 
studyabr* 
indstudy 
seniorx 
facfeed** 
envschol*** 
workhard 
memorize 
analyze 
synthesz* 
evaluate 
applying 
exams* 
readasgn 
writemor 
writemid 
writesml 
probseta* 
probsetb 



Table 5. DEVELOP Model Results 



Coefficients 



Unstandardized 

Coefficients 


Standardized 

Coefficients 


t 


m 


Collinearitv Statistics 


B 


Std. Error 


Beta 


Tolerance 


VI F 


-2.179 


.365 




-5.963 


.000 






.000 


.084 


.000 


.005 


.996 


.939 


1.065 


.073 


.115 


.015 


.634 


.526 


.876 


1.142 


.043 


.015 


.073 


2.799 


.005 


.742 


1.348 


-.136 


.039 


-.115 


-3.513 


.000 


.473 


2.114 


-.020 


.050 


-.010 


-.397 


.692 


.821 


1.219 


.011 


.023 


.017 


.494 


.621 


.405 


2.466 


.080 


.048 


.040 


1.676 


.094 


.866 


1.155 


.000 


.000 


-.073 


-2.657 


.008 


.665 


1.504 


.075 


.035 


.062 


2.107 


.035 


.577 


1.732 


-.103 


.033 


-.094 


-3.107 


.002 


.547 


1.828 


.052 


.030 


.048 


1.708 


.088 


.640 


1.563 


.060 


.035 


.050 


1.726 


.085 


.612 


1.635 


.080 


.032 


.069 


2.458 


.014 


.638 


1.566 


-.031 


.026 


-.031 


-1.164 


.244 


.709 


1.410 


.112 


.027 


.107 


4.123 


.000 


.746 


1.340 


-.028 


.028 


-.025 


-1.005 


.315 


.818 


1.222 


-.029 


.028 


-.027 


-1.020 


.308 


.714 


1.401 


.054 


.029 


.048 


1.852 


.064 


.752 


1.330 


.003 


.029 


.003 


.110 


.912 


.689 


1.451 


.006 


.033 


.006 


.171 


.864 


.434 


2.305 


.110 


.034 


.109 


3.227 


.001 


.443 


2.259 


.039 


.022 


.044 


1.784 


.075 


.842 


1.188 


-.042 


.032 


-.037 


-1.346 


.179 


.652 


1.533 


-.003 


.031 


-.003 


-.093 


.926 


.657 


1.521 


.081 


.030 


.069 


2.656 


.008 


.756 


1.323 


.030 


.024 


.032 


1.271 


.204 


.792 


1.262 


.024 


.029 


.020 


.829 


.407 


.852 


1.173 


.055 


.024 


.057 


2.331 


.020 


.857 


1.167 


-.003 


.026 


-.003 


-.115 


.908 


.835 


1.198 


.040 


.026 


.038 


1.543 


.123 


.816 


1.226 


.090 


.031 


.078 


2.921 


.004 


.705 


1.418 


.100 


.030 


.082 


3.295 


.001 


.808 


1.237 


.033 


.031 


.030 


1.081 


.280 


.642 


1.556 


.024 


.026 


.022 


.913 


.361 


.901 


1.110 


-.063 


.038 


-.049 


-1.649 


.099 


.583 


1.716 


.071 


.035 


.063 


2.049 


.041 


.530 


1.888 


.101 


.032 


.094 


3.136 


.002 


.565 


1.771 


.049 


.034 


.043 


1.459 


.145 


.586 


1.705 


.047 


.022 


.056 


2.179 


.029 


.752 


1.330 


.038 


.025 


.037 


1.517 


.129 


.833 


1.201 


-.036 


.041 


-.022 


-.869 


.385 


.786 


1.272 


-.012 


.029 


-.011 


-.423 


.673 


.702 


1.424 


-.036 


.024 


-.038 


-1.503 


.133 


.778 


1.285 


-.057 


.023 


-.065 


-2.510 


.012 


.764 


1.308 


.017 


.021 


.021 


.823 


.410 


.804 


1.244 
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Table 6. SAMECOLL Model Results 

Parameter Estimates 





Estimate 


Std. Error 


Wald 


df 


Sig. 




Lower Bound 


Upper Bound 


Threshold [samecoll = 1] 


.589 


.810 


.529 


1 


.467 


-.998 


2.176 


[samecoll = 2] 


2.217 


.806 


7.557 


1 


.006 


.636 


3.797 


[samecoll = 3] 


4.203 


.812 


26.759 


1 


.000 


2.610 


5.795 


Location enter 


-.225 


.185 


1.482 


1 


.224 


-.587 


.137 


enrlment 


.307 


.252 


1.483 


1 


.223 


-.187 


.801 


grades04 


.049 


.034 


2.036 


1 


.154 


-.018 


.116 


livenow* 


-.196 


.086 


5.234 


1 


.022 


-.365 


-.028 


gender 


.055 


.113 


.238 


1 


.625 


-.166 


.275 


classran 


-.061 


.052 


1.398 


1 


.237 


-.163 


.040 


minority** 


-.316 


.108 


8.624 


1 


.003 


-.527 


-.105 


aotsat* 


.001 


.000 


5.214 


1 


.022 


.000 


.002 


email* 


-.187 


.080 


5.515 


1 


.019 


-.343 


-.031 


facgrade 


-.005 


.075 


.004 


1 


.950 


-.151 


.141 


facplans 


.060 


.069 


.767 


1 


.381 


-.074 


.194 


facideas 


-.095 


.078 


1.474 


1 


.225 


-.249 


.058 


facother 


.046 


.074 


.382 


1 


.536 


-.099 


.190 


research 


-.043 


.059 


.537 


1 


.464 


-.159 


.072 


divclass 


.074 


.061 


1.474 


1 


.225 


-.046 


.194 


classgrp 


-.090 


.063 


2.074 


1 


.150 


-.213 


.033 


occgrp 


.120 


.064 


3.534 


1 


.060 


-.005 


.244 


tutor* 


.146 


.066 


4.836 


1 


.028 


.016 


.275 


oocideas 


.069 


.066 


1.105 


1 


.293 


-.060 


.197 


divrstud 


.012 


.074 


.025 


1 


.875 


-.134 


.158 


diffstu2 


.068 


.076 


.806 


1 


.369 


-.081 


.218 


learncom 


-.062 


.049 


1.614 


1 


.204 


-.158 


.034 


clquest 


-.055 


.071 


.595 


1 


.440 


-.195 


.085 


clpresen 


.071 


.069 


1.053 


1 


.305 


-.065 


.207 


commproj 


.060 


.069 


.747 


1 


.387 


-.075 


.194 


itacadem** 


.157 


.053 


8.737 


1 


.003 


.053 


.262 


intern 


-.059 


.064 


.847 


1 


.357 


-.185 


.067 


studyabr 


-.024 


.053 


.214 


1 


.644 


-.128 


.079 


indstudy 


-.039 


.058 


.438 


1 


.508 


-.153 


.076 


seniorx* 


.128 


.058 


4.838 


1 


.028 


.014 


.241 


facfeed** 


.201 


.069 


8.352 


1 


.004 


.065 


.337 


envschol** 


.216 


.068 


10.030 


1 


.002 


.082 


.350 


workhard*** 


-.248 


.070 


12.388 


1 


.000 


-.385 


-.110 


memorize* 


-.119 


.060 


3.994 


1 


.046 


-.237 


-.002 


analyze 


-.009 


.085 


.011 


1 


.916 


-.175 


.157 


synth esz 


.029 


.078 


.142 


1 


.706 


-.124 


.183 


evaluate 


.065 


.073 


.780 


1 


.377 


-.079 


.208 


applying 


.113 


.075 


2.287 


1 


.130 


-.034 


.261 


exams*** 


.344 


.049 


50.288 


1 


.000 


.249 


.440 


readasgn 


-.055 


.056 


.947 


1 


.331 


-.165 


.056 


writemor 


.004 


.093 


.002 


1 


.962 


-.178 


.187 


writemid 


.015 


.066 


.054 


1 


.816 


-.115 


.146 


writesml 


-.033 


.054 


.378 


1 


.539 


-.139 


.073 


probseta** 


-.131 


.051 


6.592 


1 


.010 


-.231 


-.031 


probsetb 


.079 


.047 


2.772 


1 


.096 


-.014 


.171 
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Table 7. ENTIREEXP Model Results 



Parameter Estimates 





Estimate 


Std. Error 


Wald 


df 


Sig. 


Threshold [entirexp = 1] 


3.328 


.853 


15.216 


1 


.000 


[entirexp = 2] 


5.641 


.852 


43.865 


1 


.000 


[entirexp = 3] 


8.712 


.873 


99.645 


1 


.000 


Location enter 


.059 


.193 


.094 


1 


.759 


enrlment 


.503 


.264 


3.634 


1 


.057 


grades04*** 


.193 


.036 


29.244 


1 


.000 


livenow 


-.057 


.089 


.416 


1 


.519 


gender 


-.070 


.115 


.367 


1 


.545 


classran* 


-.110 


.054 


4.218 


1 


.040 


minority* 


-.247 


.111 


4.959 


1 


.026 


actsat* 


.001 


.000 


3.840 


1 


.050 


email 


-.049 


.082 


.356 


1 


.551 


facgrade 


-.059 


.076 


.604 


1 


.437 


facplans* 


.179 


.070 


6.452 


1 


.011 


facideas 


-.011 


.080 


.020 


1 


.889 


facother 


-.022 


.075 


.083 


1 


.773 


research 


-.052 


.061 


.746 


1 


.388 


divclass 


.118 


.063 


3.565 


1 


.059 


classgrp 


-.095 


.064 


2.175 


1 


.140 


occgrp 


.058 


.065 


.788 


1 


.375 


tutor 


-.009 


.067 


.019 


1 


.890 


oocideas*** 


.215 


.067 


10.206 


1 


.001 


divrstud 


.095 


.077 


1.537 


1 


.215 


diffstu2 


-.073 


.079 


.856 


1 


.355 


learncom 


-.091 


.050 


3.285 


1 


.070 


clquest 


.028 


.073 


.150 


1 


.698 


clpresen 


.113 


.071 


2.520 


1 


.112 


commproj 


-.020 


.070 


.079 


1 


.779 


itacadem 


.082 


.055 


2.226 


1 


.136 


intern 


.014 


.067 


.045 


1 


.832 


studyabr 


.048 


.054 


.761 


1 


.383 


indstudy 


-.025 


.060 


.176 


1 


.675 


seniorx** 


.156 


.060 


6.772 


1 


.009 


facfeed*** 


.348 


.072 


23.553 


1 


.000 


envschol*** 


.357 


.071 


25.429 


1 


.000 


workhard* 


-.171 


.072 


5.653 


1 


.017 


memorize*** 


-.239 


.061 


15.106 


1 


.000 


analyze 


.021 


.087 


.058 


1 


.810 


synthesz 


.073 


.081 


.819 


1 


.365 


evaluate 


-.006 


.075 


.006 


1 


.939 


applying* 


.180 


.078 


5.367 


1 


.021 


exams*** 


.366 


.050 


52.673 


1 


.000 


readasgn 


.032 


.058 


.297 


1 


.585 


writemor 


.005 


.096 


.003 


1 


.956 


writemid 


.036 


.068 


.277 


1 


.599 


writesml 


-.037 


.055 


.453 


1 


.501 


probseta 


-.096 


.052 


3.390 


1 


.066 


probsetb 


.062 


.048 


1.648 


1 


.199 



95% Confidence Interval 


Lower Bound 


Upper Bound 


1.656 


cn 

o 

o 

o 


3.972 


7.311 


7.002 


10.423 


-.319 


.437 


-.014 


1.020 


.123 


.263 


-.232 


.117 


-.296 


.156 


-.215 


-.005 


-.463 


-.030 


.000 


.002 


-.209 


.111 


-.209 


.090 


.041 


.316 


-.169 


.146 


-.169 


.126 


-.171 


.066 


-.005 


.241 


-.220 


.031 


-.070 


.186 


-.141 


.122 


.083 


.347 


-.055 


.245 


-.227 


.081 


-.190 


.007 


-.115 


.171 


-.027 


.253 


-.157 


.117 


-.026 


.189 


-.116 


.145 


-.059 


.154 


-.143 


.093 


.038 


.273 


.207 


.488 


.218 


.495 


-.311 


-.030 


-.359 


-.118 


-.150 


.192 


-.085 


.231 


-.152 


.141 


.028 


.332 


.267 


.465 


-.082 


.145 


-.182 


.193 


-.097 


.169 


-.146 


.071 


-.199 


.006 


-.033 


.157 
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DISCUSSION AND IMPLICATIONS 

There are a number of interesting and significant findings that emerge from the 
regression results. With notable consistency and significance across all four models, the 
extent to which exams challenge students to do their best work, institutional emphasis on 
academics and studying, and the frequency of prompt feedback on academic performance 
had a positive relationship to LRNGAIN, DEVELOP, SAMECOLL, and ENTIREEXP. 
These results appear to support the view that efforts to increase institutional impact on 
student learning and development and increasing student satisfaction do not work at cross 
purposes. In fact, these two “competing” strategic views of how best to enhance 
academic quality, institutional, and persistence actually appear to be supported by and 
related to the same set of institutional foci and practices. These findings may suggest 
some interesting implications for enhancing the communication and level of expectations 
to students, exploring the motivation and aspirations of students who attend, and 
improving methods to gauge and reward instructional quality based on the level of 
academic challenge and the quality of faculty feedback to students. However, the 
apparent negative relationship between the amount of short homework assignments 
(PROBSETA) and DEVELOP and SAMECOLL also suggests that programs and courses 
that employ such an approach to student preparation and evaluation may benefit from 
exploring a more diverse set of course experiences and requirements (i.e. application- 
focused assignments, problems and projects; small study groups; integrated technology) 
as suggested by other findings in this study. 

These results are supported further by the apparent role of emphasis on higher- 
order critical thinking skills in courses. The relationship is most prominent in the 
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LRNGAIN model where synthesis (SYNTHESZ) and application (APPLYING) are 
significant at the .001 level and making judgments (EVALUATE) was significant at the 
.01 level. SYNTHESZ also was significant in the DEVELOP model. In a 
complementary fashion, course emphasis on “memorization to repeat information back in 
the same form” (MEMORIZE) had a negative relationship with SAMECOLL and 
ENTIREEXP. 

In addition — and contrary to what one might expect — peer interactions (OCCGRP) 
and conversations about course ideas and readings with family and others (OCCIDEAS) 
appear to have a stronger relationship with LRNGAIN in comparison to faculty 
member/instructor interactions and faculty/instructor interactions appear to be more 
strongly related to DEVELOP. Strong peer interactions within the context of classroom 
learning and strong faculty interactions outside of the classroom may productively 
challenge students’ initial expectations of faculty and peer roles in and out of the 
classroom. In fact, serving as a tutor for peers (TUTOR) - another form of peer 
interaction — also had a positive relationship with SAMECOLL. In other words, creating 
a quality undergraduate education is a collective responsibility and this includes students’ 
responsibility to their peers in classroom and class-related settings and faculty members’ 
responsibility to students outside of the classroom. However, the results also suggest that 
interaction between students and faculty may be most productive when it is regular, 
prompt, and focused on academic performance (FEEDBACK). This finding is supported 
by Kuh and Hu (2001b). 

The interesting relationship between academic preparation and the dependent 
variables in all four models also is worth noting. Specifically, the level of academic 
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preparation as measured by ACTSAT had an insignificant relationship with LRNGAIN, a 
negative relationship with DEVELOP, and a positive relationship with SAMECOLL and 
ENTIREEXP. These findings may suggest that students who enter college with higher 
levels of academic preparation - given their relatively higher “starting point”- may not 
experience as much change during their college years even as they are more likely to be 
satisfied with their experience and persist. This finding is especially important to keep in 
mind when and where institutions or policymakers pursue “value added” approaches to 
institutional effectiveness. It is likely that more selective institutions (institutions with 
better-prepared students) may not appear to be as effective on the basis of such an 
approach. Even so, it does not appear that academic preparation or selectivity is the most 
important indicator of institutional impact or satisfaction across the various quality 
dimensions represented by these four models. 

Specific elements of academic and institutional programming and the curriculum also 
appeared to have a positive and significant relationship with various dimensions of 
institutional impact and student satisfaction as well. In particular, the use of technology 
in courses (ITACADEM) had a positive and significant relationship with LRNGAIN and 
SAMECOLL. This result lends additional support to recent findings by Nelson-Laird 
(2005) and suggests that investments in technological infrastructure and training are a 
productive use of resources (Ryan, 2004 and 2005; Smart, Ethington, Riggs, and 
Thompson, 2002; Toutkoushian and Smart, 2001). At the same time, there was a 
negative and significant relationship between using e-mail to communicate with an 
instructor (EMAIL) and SAMECOLL. The reasons for using e-mail, subject matter, and 
preferences of the student and instructor may be key determinants behind this 
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relationship. Students that prefer personal communication and do not feel they have that 
opportunity, the use of e-mail as a way to raise problems or difficult issues face-to-face, 
and the effects of e-mail as a mode of communication that confirms expectations of a 
large, impersonal environment and contributes to social isolation represent some possible 
explanations. 

Additional opportunities or embedded requirements to complete a culminating senior 
experience (SENIORX) and study abroad experiences (STUDYABR) also had positive 
and significant relationships with LRNGAIN, SAMECOLL, and ENTIREEXP and with 
DEVELOP respectively. The benefits of service learning (Jones, 2003a, 2003b, 2004) 
were supported as well. When considered in tandem with the potential benefits of 
student-faculty discussions about career plans (FACPLANS) and student conversations 
with peers with different beliefs, opinions, and values in and out of class (DIFFSTU2 and 
DIVCLASS), attempts to link theory or concepts learned in class with their role and 
application in “real” settings with others who have different perspectives and experiences 
appear to play important roles and warrant serious consideration by institutional leaders 
in their planning and programming decisions. These results also support the role that out 
of class experiences and their positive relationship with desired outcomes (Kuh, 1995). 

Contrary to what accrediting agencies typically suggest (Higher Learning 
Commission, 2004), grades may serve as a valid proxy indicator of academic quality and 
impact and can play a meaningful role in fostering student outcomes assessment within 
the institutional culture (Palomba and Banta, 1999; Walvoord, 2004). Given the long- 
standing practice of grading as a means of documenting and communicating achievement 
in a course, engaging faculty and constituents who seek to enhance accountability around 
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defining and “unpacking” the meaning of grades may provide a starting point for 
discussing course and program improvement. 

The models also produced what appear to be contradictory and unexpected results. 
EXAMS and WORKHARD exhibited opposing relationships in both the SAMECOLL 
and ENTIREXP models. What might explain this apparent contradiction and distinction 
between being challenged to do one’s best and working harder than one thought was 
possible to meet an instructor’s expectations? This result may reflect the prior academic 
experience of students, particularly in high school. For many years, the Higher Education 
Research Institute (HERI) Freshman Survey results has suggested a trend of higher 
reported grades and low reported effort to achieve those grades among college freshmen 
from across the United States (HERI, 2003). Students who come from backgrounds with 
a combination of low effort and high grade performance may have a more difficult time 
making the transition to college and adjusting to the higher amount of effort required for 
similar levels of grade performance. 

Another unexpected result centers on the insignificant relationship between 
research experiences in all four models. One might expect such experiences to play an 
important role given its place and strength at research-extensive universities. It is 
possible that such experiences may be captured by responses to SENIORX and 
FACOTHER. However, it also may reflect the need to give more attention and improved 
coordination to enhance the role of research experiences in undergraduate education and 
work to challenge the view that research and undergraduate instruction represent 
competing agendas in a zero-sum game. 
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Among the remaining control variables, MINORITY was insignificant in the 
LRNGAIN and DEVELOP models. However, it exhibited a negative relationship with 
SAMECOLL and ENTIREEXP. This suggests that more work needs to be done not only 
to seek fair levels of diversity and representation but also to enhance the supportive and 
welcoming aspects of the institutional environment that improve minority students’ 
overall experience once they matriculate. This is particularly important given the 
remaining gap in degree attainment between minority and non-minority students, the 
particular challenges to degree attainment for African-American males (American 
Council for Education, 2003), and the strong, positive relationship between DIFFSTU2 
and DEVELOP suggested by the results in this study. The impact of diversity and access 
policies and the quality of the educational experience on the representation of students 
from different backgrounds in the post-9/1 1 and post-Grutter v. Bollinger era will need to 
be watched very closely if both equal treatment (access) and the affective development of 
students are to maintain their status as principles of an effective post- secondary 
educational system. 

Living off-campus also had a negative relationship with DEVELOP. Those who live 
on or at least closer to campus may benefit more from access to and involvement with the 
campus community. Students who live away from campus also may be more likely to 
experience other pressures or responsibilities such as more hours per week in a paid 
position and dependent care. Campus housing options that accommodate different 
living/family needs, services that minimize the impact of transportation and dependent 
care, and programs tailored to enhance “quality time” with the institution may require 
increased attention and creative thinking on the part of institutions and policymakers. 
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LIMITATIONS 

Although this study points to a set of interesting findings, especially for large, 
research-extensive institutions, there are some limitations to consider. 

First, this study was conducted based on a single institution at a single point in 
time. Other studies of large institutions and research-extensive institutions (and 
institutions of other types) will be required to continue down the path of determining 
whether or not particular forms of engagement and best practices and their relationships 
to various institutional and student outcomes differ across institution types. Also, the 
NSSE survey is administered in the spring; at that point in the academic year, there is 
some number of students (notably first-year students) who have already left the 
institution and taken a potentially important and distinct source of information along 
with them. Future attempts to explore these missing cases might be a fruitful line of 
inquiry. 

In addition, there can be differences between respondents and non-respondents 
(Porter and Whitcomb, 2005). In this study, students who participated and had complete 
data were more likely to be white, female, live on campus, non-transfers, and enrolled 
full-time (based on chi-square and Mann-Whitney tests of mean rank differences). 

Another limitation involves the role of student personality (Dollinger, 2000), 
motivation, and other background and institutional variables that may lead to omitted- 
variable bias in the models. Although variables selection was guided by theory and prior 
research, the integration of additional components found within psychological, social, 
institutional, and economic models could enhance the explanatory power of “institutional 
impact” models such as those used in this study. Efforts to integrate key components of 
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different perspectives into a more unified model may help to unite various research 
programs and agendas while providing a richer context for examining a complex process. 
Here, mixed methods approaches may help to connect model components and explore 
casual and temporal links within a more unified “quality” or “institutional impact” model 
of undergraduate education (Cabrera and Castenada, 1993). These models also might 
include components of research productivity and outreach to further integrate and better 
identify leverage and sticking points across various outcome and mission areas. 

Lastly, it is important to consider unique aspects of the institutional context. The 
institution used as the focus for this study has pursued and realized an increasing 
educational profile among its undergraduate students. Entering freshman class ACT 
averages have increased steadily and considerably in recent years. It is possible that this 
trend and dynamic may play a role in explaining the positive relationship between 
academic challenge and institutional emphasis on academics with LRNGAIN, 
DEVELOP, SAMECOLL, and ENTIREEXP. The institution may need to adjust its 
expectations and identity to “catch up” with the changing needs and expectations of the 
students they teach and mentor. 

CONCLUSION 

To summarize these findings within the context of the Chickering and Gamson 
framework and the main research questions, the overall results suggest that high 
expectations and prompt feedback on academic performance may provide the most 
fruitful avenues for improving undergraduate education outcomes at large, research- 
extensive universities. Specific elements and kinds of engagement may “matter” more 
than others and specific engagement items may hold their place of importance across a 
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range of institutional and student outcomes of interest, suggesting a certain link between 
best practices that enhance learning and development and best practices that enhance 
student satisfaction, retention, and perceptions of the quality of their overall educational 
experience. However, it is clear that specific kinds of contact with peers and faculty and 
enriching educational experiences (or diverse learning opportunities) play an important 
role as well. 

Also, student academic preparation — although significant in three of the four 
models — does not appear to be the most important factor when institutional practices 
and student experiences are taken into account. What happens to students and what 
students do during their collegiate experience appears to matter even more. As a result, 
institutional initiatives and public policies that seek to enhance instruction and learning 
via specific educational experiences and thoughtful integration of technology (Barr and 
Tagg, 1995; Guskin, 1994a and 1994b; Twigg, 2003) - with the funding to create and 
sustain them - may be productive ways to channel valuable resources. 

For institutions with complex missions and large scale operations in instruction, 
research, and public service, the application of sound theory and empirical methods of 
inquiry to shape priorities and identify key leverage points to improve undergraduate 
education represents an approach that brings well-established principles of scholarship 
and the benefit of research to bear on the critical challenge of improving undergraduate 
education. Further inquiry and study focused on individual institutions by both 
researchers and leaders along the lines suggested by this study appears warranted. 
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